Kafka consists of Records, Topics, Consumers, Producers, Brokers, Logs,
Partitions, and Clusters. Records can have key (optional), value and
timestamp. Kafka Records are immutable. A Kafka Topic is a stream of
records (
"/orders", "/user-signups"). You can
think of a Topic as a feed name. A topic has a Log which is the topic’s
storage on disk. A Topic Log is broken up into partitions and segments.
The Kafka Producer API is used to produce streams of data records. The
Kafka Consumer API is used to consume a stream of records from Kafka. A
Broker is a Kafka server that runs in a Kafka Cluster. Kafka Brokers
form a cluster. The Kafka Cluster consists of many Kafka Brokers on many
servers. Broker sometimes refer to more of a logical system or as
Kafka as a whole.
The Three Main Component
- Producer
- Broker
- Consumer
The producers are the client applications, and they send some messages.
The Brokers receive those messages from publishers and store them.
The consumers read the message records from brokers.
The Brokers receive those messages from publishers and store them.
The consumers read the message records from brokers.
Kafka Producers
Producers in Kafka push data to brokers. Also, all the producers search it and automatically sends a message to that new broker, exactly when the new broker starts. However, keep in mind that the Kafka producer sends messages as fast as the broker can handle, it doesn’t wait for acknowledgments from the broker.
Producers in Kafka push data to brokers. Also, all the producers search it and automatically sends a message to that new broker, exactly when the new broker starts. However, keep in mind that the Kafka producer sends messages as fast as the broker can handle, it doesn’t wait for acknowledgments from the broker.
Kafka Consumers
Basically, by using partition offset the Kafka Consumer maintains that how many messages have been consumed because Kafka brokers are stateless. Moreover, you can assure that the consumer has consumed all prior messages once the consumer acknowledges a particular message offset. Also, in order to have a buffer of bytes ready to consume, the consumer issues an asynchronous pull request to the broker. Then simply by supplying an offset value, consumers can rewind or skip to any point in a partition. In addition, ZooKeeper notifies Consumer offset value
Basically, by using partition offset the Kafka Consumer maintains that how many messages have been consumed because Kafka brokers are stateless. Moreover, you can assure that the consumer has consumed all prior messages once the consumer acknowledges a particular message offset. Also, in order to have a buffer of bytes ready to consume, the consumer issues an asynchronous pull request to the broker. Then simply by supplying an offset value, consumers can rewind or skip to any point in a partition. In addition, ZooKeeper notifies Consumer offset value
Broker
An instance in a Kafka cluster is called a broker. In a
Kafka cluster, if you connect to any one broker, you will be able to
access the entire cluster. The broker instance that we connect to in
order to access the cluster is known as a bootstrap server. Each broker is identified by a numeric ID in the cluster. To start a Kafka cluster, three brokers is a good number, but there are clusters with hundreds of brokers.
Topic
A topic is a logical name to which the records are
published. Internally, the topic is divided into partitions to which the
data is published. These partitions are distributed across the brokers
in the cluster. For example, if a topic has three partitions with three
brokers in the cluster, each broker has one partition. The published
data to partition is append-only with the offset increment.
For the purpose of managing and coordinating, Kafka broker uses ZooKeeper. Also, uses it to notify producer and consumer about the presence of any new broker in the Kafka system or failure of the broker in the Kafka system. As soon as Zookeeper send the notification regarding presence or failure of the broker then producer and consumer, take the decision and starts coordinating their task with some other broker.
No comments:
Post a Comment